Add replace command with Calcite by manasvinibs · Pull Request #4248 · opensearch-project/sql

manasvinibs · 2025-09-08T16:42:59Z

Description

Implement the replace command in PPL to replace text patterns in specified fields. This PR includes the grammar implementation and basic replacement functionality. This implementation reuses the existing replace functionality. Missing/new features for regex support is added in a separate PR.

Syntax

<source> | replace '<pattern>' WITH '<replacement>' IN field_list

pattern: The text pattern to search for (case-sensitive)
replacement: The text to replace matches with
field_list: Comma-separated list of fields to perform replacement in
Replace creates new fields with prefix 'new_' containing the replaced text, while preserving original fields.

Semantics

Expected Behavior:

Action: Creates new fields with replaced text values
Scope: Operates only on specified fields
Data Preservation: Original fields remain unchanged
Case Sensitivity: Text literal matching is case-sensitive
Pattern Type: Only supports literal string patterns (wildcards or regex will be added in a separate PR)

Implementation Approach:

Creates new fields with 'new_' prefix for each specified field
Performs literal string replacement in the specified fields
Maintains original field values
Validates field existence and pattern/replacement values

Example Queries

-- Replace in single field
source=logs | replace 'error' WITH 'ERROR' IN message

-- Replace in multiple fields 
source=logs | replace 'USA' WITH 'United States' IN country, state

-- Replace with other commands
source=logs | where level='error' | replace 'error' WITH 'ERROR' IN message | sort @timestamp

-- Replace and select specific fields
source=logs | replace 'error' WITH 'ERROR' IN message | fields message, new_message

Output Schema

For each field specified in IN clause, a new field is created:

Original field: remains unchanged
New field: prefixed with 'new_' containing replaced text

Example:

Input fields: message="error occurred", level="error"
After replace 'error' WITH 'ERROR' IN message, level:

message: "error occurred"
new_message: "ERROR occurred"
level: "error"
new_level: "ERROR"

Related Issues

#3975

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Swiddis · 2025-09-08T16:53:49Z

core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

+  @Override
+  public LogicalPlan visitReplace(Replace node, AnalysisContext context) {
+    throw new UnsupportedOperationException(
+        "Replace is supported only when " + CALCITE_ENGINE_ENABLED.getKeyValue() + "=true");


There's a util for this nowadays:

sql/core/src/main/java/org/opensearch/sql/calcite/utils/CalciteUtils.java

Line 15 in c186686

public static UnsupportedOperationException getOnlyForCalciteException(String feature) {

Swiddis · 2025-09-08T16:56:22Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+    String patternStr = pattern.toString();
+    if (patternStr.contains("+")
+        || patternStr.contains("-")
+        || patternStr.contains("*")
+        || patternStr.contains("/")) {
+      throw new IllegalArgumentException(
+          "Expression is not allowed in replace pattern. Only string literals are supported.");


A little confused by this logic, we don't allow strings that contain these characters? Or if we're trying to forbid any expressions at all, I'm not sure just the 4 arithmetic operators will be sufficient (e.g. if someone tries to boolean and or something).

If we don't want an expression, the grammar should specify a stringLiteral or something along those lines, instead of expression.

+1 for @Swiddis 's question, and I also left a comment in this validation method in general.

Thanks for pointing this! Yes the idea was to avoid any non-string literals for pattern text. I have updated the grammar token itself to string literal to handle this use case.

RyanL1997

Hi @manasvinibs , thanks for taking this on. And I just left some comments.

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

RyanL1997 · 2025-09-08T21:19:18Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+  }
+
+  private void validate() {
+    if (pattern == null) {


The validation logic for the pattern expression hardcodes checks for mathematical operators (+, -, *, /) on L44-47. This is may miss other invalid expressions in my opinion. Maybe consider validating that pattern is a string literal by checking its type (e.g., Literal with DataType.STRING)?

Something like this:

if (!(pattern instanceof Literal && ((Literal) pattern).getType() == DataType.STRING)) { throw new IllegalArgumentException("Replace pattern must be a string literal."); }

Good call! Updated in the next revision.

RyanL1997 · 2025-09-08T21:19:57Z

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

+    String patternStr = pattern.toString();
+    if (patternStr.contains("+")
+        || patternStr.contains("-")
+        || patternStr.contains("*")
+        || patternStr.contains("/")) {
+      throw new IllegalArgumentException(
+          "Expression is not allowed in replace pattern. Only string literals are supported.");


+1 for @Swiddis 's question, and I also left a comment in this validation method in general.

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

ppl/src/main/antlr/OpenSearchPPLParser.g4

ykmr1224

Wild card support for rename command is implemented in #4019
Let's utilize the logic when we implement wild card support for replace.

docs/user/ppl/cmd/replace.rst

ppl/src/main/antlr/OpenSearchPPLParser.g4

ykmr1224 · 2025-09-09T22:41:22Z

docs/user/ppl/cmd/replace.rst

+
+Description
+============
+| Using ``replace`` command to replace text in one or more fields in the search result. The replaced text appears in new fields with prefix *new_*.


Is it our intended behavior to introduce new attribute?

Currently with this implementation I'm reusing existing eval replace function which also exposes the replaced values into new fields. I'm mimicking the same behavior with direct command. But, if we think instead of adding new field, we should replace on the original one I'm open to explore that option. Let me know your thoughts.

docs/user/ppl/cmd/replace.rst

ykmr1224 · 2025-09-09T22:44:02Z

docs/user/ppl/cmd/replace.rst

+============
+replace '<pattern>' WITH '<replacement>' IN <field-name>[, <field-name>]...
+
+* pattern: mandatory. The text pattern you want to replace.


Let's clarify that we currently support only plain text pattern. (pattern could indicate wildcard/regex/etc.)

ykmr1224 · 2025-09-09T22:45:12Z

docs/user/ppl/cmd/replace.rst

+ fetched rows / total rows = 1/1
+ +-------------------------------+-------------------------------+
+ | message                       | new_message                   |
+ |------------------------+--------------------------------------|


nit: I see broken table format.

ykmr1224 · 2025-09-09T22:47:08Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+      newFieldNames.add(fieldName);
+    }
+
+    // Then add new fields with replaced content using new_ prefix


What happens if new_xxx already exist?

I tested this behavior and I see system is automatically handling the field name collision by appending a number ("new_country0") and resolving field name conflicts gracefully. I have documented this behavior.

docs/user/ppl/cmd/replace.rst

core/src/main/java/org/opensearch/sql/ast/tree/Replace.java

ykmr1224 · 2025-09-16T18:31:04Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+          context.relBuilder.call(
+              SqlStdOperatorTable.REPLACE, fieldRef, patternNode, replacementNode);
+      projectList.add(replaceCall);
+      newFieldNames.add(NEW_FIELD_PREFIX + fieldName);


Looks it does not check existing field and add suffix number as written in the doc.

Yes every replace command adds a new_ column and do not conflict with existing column names. Let me know if you think we should change the behavior.

I was wondering the following doc description is right since this logic doesn't check existence of the field. (Is it something automatically done?)

* If a field with *new_* prefix already exists (e.g., 'new_country'), a number will be appended to create a unique field name (e.g., 'new_country0')

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java

ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java

Signed-off-by: Manasvini B S <manasvis@amazon.com>

opensearch-trigger-bot · 2025-10-14T15:22:15Z

This PR is stalled because it has been open for 2 weeks with no activity.

Swiddis reviewed Sep 8, 2025

View reviewed changes

RyanL1997 reviewed Sep 8, 2025

View reviewed changes

ykmr1224 reviewed Sep 9, 2025

View reviewed changes

docs/user/ppl/cmd/replace.rst Show resolved Hide resolved

manasvinibs force-pushed the replace-ppl branch from ef65df6 to 2d17ce7 Compare September 11, 2025 16:14

manasvinibs requested a review from ykmr1224 September 11, 2025 16:35

manasvinibs force-pushed the replace-ppl branch from 2d17ce7 to be806e8 Compare September 11, 2025 16:44

manasvinibs added the enhancement New feature or request label Sep 11, 2025

manasvinibs added PPL Piped processing language calcite calcite migration releated labels Sep 11, 2025

manasvinibs force-pushed the replace-ppl branch 6 times, most recently from a20ff96 to 966b939 Compare September 11, 2025 20:32

ykmr1224 reviewed Sep 16, 2025

View reviewed changes

manasvinibs mentioned this pull request Sep 18, 2025

Add wildcard support for replace command manasvinibs/sql#1

Draft

Add replace command with Calcite

3621bc9

Signed-off-by: Manasvini B S <manasvis@amazon.com>

manasvinibs force-pushed the replace-ppl branch from 966b939 to 3621bc9 Compare September 19, 2025 00:15

ahkcs mentioned this pull request Oct 7, 2025

Add replace command with Calcite #4451

Merged

8 tasks

opensearch-trigger-bot bot added the stalled label Oct 14, 2025

RyanL1997 closed this Oct 28, 2025

Conversation

manasvinibs commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Syntax

Semantics

Expected Behavior:

Implementation Approach:

Example Queries

Output Schema

Example:

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RyanL1997 Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RyanL1997 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RyanL1997 Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ykmr1224 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

manasvinibs commented Sep 8, 2025 •

edited

Loading

Swiddis Sep 8, 2025 •

edited

Loading

RyanL1997 Sep 8, 2025 •

edited

Loading

RyanL1997 Sep 8, 2025 •

edited

Loading